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(57) A retouching method for enhancing an appear- 
ance of a face located in a digital image involves acquir- 
ing a digital image containing one or more faces and 
detecting a location of facial feature points in the faces, 
where the facial feature points include points identifying 
salient features such as skin, eyes, eyebrows, nose, 
mouth, and hair. The location of the facial feature points 
are used to segment the face into different regions such 
as skin, eyes, eyebrows, nose, mouth, neck and hair re- 
gions. Facially relevant characteristics of the different 
regions are determined and, based on these facially rel- 
evant characteristics, an ensemble of enhancement fil- 
ters are selected, each customized especially for a par- 
ticular region, and the default parameters for the en- 
hancement filters are selected. The enhancement filters 
are then executed on the particular regions, thereby pro- 
ducing an enhanced digital imagef rom the digital image. 
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Description 

FIELD OF THE INVENTION 

s [0001] The present invention relates generally to the field of digital image processing, and in particular to the creation 
of improved imaging products derived from portrait-type images of human subjects. 

BACKGROUND OF THE INVENTION 

io [0002] For centuries, only the wealthy or privileged classes of society could afford to employ the skilled artisans who 
labored to produce a fine likeness in painting, sculpture, and drawing. In many cases, portraiture served a purpose 
greater than the simple creation of an acceptable likeness of reality. In subtle or overt ways, the artists worK would 
interact with the desires and intentions of the subjects. A second category of artistic license involved improvement on 
reality -. Thus, subjects were rendered in such a way as to minimize their physical imperfections and to present the most 
• 5 attractive possible appearance. 

[0003] In modem society, portraiture is no longer the exclusive domain of the wealthy and powerful The advent of 
photography into all levels of society has rendered creation of portrait images to be an ubiquitous part of many of life's 
major events. Weddings, graduations, birthdays, arrival of a baby - all of these events, and more - are commonly 
captured with relatively standardized portrait images in western cultures. While the state of technology enables indi- 
vidual amateurs to capture and even enhance images such as these, there still exists a class of professional photog- 
raphers that provide creation of higher-quality portrait images. Not surprisingly, the goals of the portraitist remain the 
same as in bygone centuries - to present the subject in the most pleasing possible way. In essence, the subject wants 
to be seen as they wish they were, not as they really are. 

[0004] In response to the desire for people to be seen as they wish they are, and not the way they really are pro- 
fessional photographers resort to retouching the portrait image to give people their preferred appearance. Retouching 
involves changing a photo image in some way that was not captured or depicted in the original photographic image 
One of the goals of retouching a portrait image is to make a person look better by removing temporary imperfections 
such as blemishes or dark circles under the eyes or permanent imperfections such as moles or wrinkles while still 
maintaining the personality of the individual. Removing facial blemishes, moles and scars, softening lines and wrinkles 
decreasing bags under the eyes, whitening teeth and the whites of the eyes are examples of retouching performed to 
improve or enhance the appearance of an individual in a portrait image. 

[0005] Before the advent of the digital age, retouching of images were performed on either the negative or printed 
image by modifying the image using dyes to mask or change imperfections in the portrait image. Now that digital imaqe 
capture devices are routinely available, the preferred method of retouching is done via digital imaging techniques 
performed on the captured digital images. Digital methods allow enhancements to be performed that were either ex- 
ST V k !2 t0 previous, y im P° ss 'ble to perform on the analogue image. Image editing software such as Adobe 
digSmate-uT * rem ° Vin9 blemisnes - straightening noses, balancing eyes and applying 

[0006] Improvements in computer technology and image processing algorithms are enabling new classes of auto- 
mated and semi-automated image enhancements. Relating to the subject of portrait images, relevant technological 
developments include face detection and recognition, facial feature detection and masking, face re-posinq and red- 
eye detection and correction. a ' 
[0007] in published PCT Patent Application WO 00/76398 A1 , "Skin Imaging Analysis Systems and Methods" Mill- 
ebrand et al. disclose a system that can detect skin defects and calculate a skin severity index. This system is aimed 
towards he cosmetic and skin care market. The system can also simulate improvements to the defected skin areas 
that would be realized upon the use of a recommended treatment or product that eliminates or hides the skin defect 
The skin defects are detected using color information and standard morphing techniques are used to simulate improve- 
ments in the defected skin areas, i- 

[0008] m published European Patent Application EP 1 030 276 Al, "Method of Correcting Face Image, Makeup 
Simulation Method, Makeup Method, Makeup Supporting Device and Foundation Transfer Film", Utsugi describes a 
method for preparing an ideal post-makeup face through image processing based on a desirable face or a model face 
The technique involves making highlighted areas, and the shapes of the eyebrows, the eyelines, and the lip line closer 
to that of a desirable face or a model face within a range where modification by makeup is possible 
[0009] The drawback of such systems, especially for batch portraiture systems as used, e.g. , for weddings gradu- 
ations, school and sports pictures, birthdays, arrival of a baby, etc., is the intense interaction required with the customer 
to input preferences and evaluate results. For example, in Utsugi the makeup customer's presence is required to settle 
on the model face, e.g., selected from the faces of popular talents, actresses or actors, and on the various adjustments 
made to reach the model face. Moreover, a skilled operator is required to work with the customer to produce an ac- 
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ceptable result. 

[0010] Even with the advent of digital imaging, therefore, retouching portraits is a craft unto itself and to this day 
remains more of an art form than a science. In addition, the process of retouching portrait images is a highly manual 
and time consuming process performed by skilled operators. It therefore would be advantageous to develop a system 
5 that uses automated and semi-automated portrait image enhancement methods to enable the facile retouching of 
portraits. The present invention solves the above mentioned shortcomings of the current art by providing methods and 
system for automated enhancement of the appearance of the human subjects in images. 

SUMMARY OF THE INVENTION 

10 

[0011] The present invention is directed to overcoming one or more of the problems set forth above. Briefly summa- 
rized, according to one aspect of the present invention, a retouching method for enhancing an appearance of a face 
located in a digital image comprises the steps of: (a) acquiring a digital image containing one or more faces: (b) detecting 
a location of facial feature points in the one or more faces, said facial feature points including points identifying salient 

15 features including one or more of skin, eyes, eyebrows, nose, mouth, and hair; (c) using the location of the facial feature 
points to segment the face into different regions, said different regions including one or more of skin, eyes, eyebrows, 
nose, mouth, neck and hair regions; (d) determining one or more facially relevant characteristics of the different regions; 
(e) based on the determined facially relevant characteristics of the different regions of the face, selecting (1) two or 
more enhancement filters each customized especially for a particular region and (2) the default parameters for the 

20 enhancement filters; and (f) executing the enhancement filters on the particular regions, thereby producing an enhanced 
digital image from the digital image. 

[0012] The advantage of the invention is that it efficiently uses automated and semi-automated portrait image en- 
hancements methods to enable the retouching of portraits without requiring skilled operator intervention to make and 
supervise the retouching corrections. Thus, the highly manual and time consuming processes performed by skilled 
25 operators is avoided and the retouching method may be implemented on a batch process. 

[0013] These and other aspects, objects, features and advantages of the present invention will be more clearly 
understood and appreciated from a review of the following detailed description of the preferred embodiments and 
appended claims, and by reference to the accompanying drawings. 

30 BRIEF DESCRIPTION OF THE DRAWINGS 

[0014] I 

FIGS. 1 A and 1 B are diagrams of a system for practicing the invention and a graphical user interface for use with 
35 the system, respectively. 

FIG. 2A is a flowchart of an embodiment for implementing the invention with an optional degree of user intervention . 
FIG. 2B is a flowchart of an automatic embodiment for implementing the invention in a batch mode without user 
intervention. 

FIG. 3 is a flowchart of the facial detection and facial point location steps shown in FIGS. 2A and 2B. 
40 FIG. 4 is a pictorial example showing the location of salient feature points on a detected face. 

FIG. 5A is a flowchart of a method for determining a neck region. 

FIGS. 5B - 5D are pictorial examples that visually illustrate the process of creating the final neck probability map. 
FIG. 6 is a diagram of the ensemble of enhancement filters used in the system illustrated in the flowcharts of FIGS. 
2A and 2B. 

45 FIG. 7A is a flowchart for a skin texture enhancing filter shown in Figure 6. 

FIG. 7B is a diagram of a pixel neighborhood comprising the valley edge filter kernal used by the skin texture 
enhancing filter. 

FIG. 8 is a flow chart showing a preferred embodiment of selecting and modifying skin features according to the 
invention. 

so FIG. 9 is a pictorial example of a pinwheel filter illustrating line segments defined around a defect pixel. 

FIGS. 1 0A and 1 0B together represent a flow chart illustrating a process utilizing the pinwheel filter of Figure 9 for 
estimating corrected values for a defect pixel. 

FIG. 11 is a flowchart illustrating a preferred embodiment for calculating new pixel values in the process illustrated 
in Figure 10. 

55 FIG. 12 is a flow chart illustrating a process for creating line segments through a feature pixel in the pinwheel filter 

shown in Figure 9. 

FIG. 1 3 is a flow chart of a presently preferred embodiment of the skin tone enhancing filter as shown in Figure 6. 
FIG. 1 4 is a flow chart of a presently preferred embodiment of the calculation of shadow/highlight strength for use 
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in the skin tone enhancing filter shown in Figure 13. 

FIGS. 1 5A, 1SB and 1 5C are illustrations of several functions showing the effect of different blending coefficients 
used in the skin enhancing filter shown in Figure 13. 

FIG. 16 is a flow chart of a presently preferred embodiment of the teeth and eye enhancing filters shown in Figure 6 
FIG. 1 7 is an illustration of a blending function used in the skin texture enhancing filter. 
FIG. 18 is a pictorial example of the control points used in making a shape adjustment of an eye 
FIG. 19 is a flowchart for a presently preferred "embodiment of a shape enhancing filter as shown in Figure 6. 

DETAILED DESCRIPTION OF THE INVENTION 

[0015] Because image processing systems employing facial feature detection and feature enhancement are well 
fT 1 d , eSCripti0n Wi " be directed in P articular to attributa * Arming part of, or cooperating more directly 
mav STSiSj 7 T acCOrdance wi,h the P resent inven «on. Attributes not specifically shown or described herein 
may be selected from those known in the art. In the following description, a preferred embodiment of the present 
invention wou.d ordinarily be implemented as a software program, although thosesklHed in the art wil. readily rlcTgn'e 
that the equivalent of such software may also be constructed in hardware. Given the system as described according 
or TmZ!!2rt th f t ! 0 " OWina materials - software not specifically shown, suggested or described herein that is useful 
fo implementation of the invention is conventional and within theordinary skill insuch arts. As a matter of nomenclature 
~-^" P T, P T 6nt inVenli ° n ' there iS ref6renCe 10 ennanc ^ent fillers as methods that enhance the visuai 
aXTns^ 

SoI!L ' f t the inVen, !° n iS irn P' emented as a com P"ter program, the program may be stored in conventional computer 

Z ^ D vSThS r may comprise ' f or example; magnetic stora9e media such as a ma 9 netic dfsk 

as a floppy disk or a hard drive) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine 
readable bar code; solid state electronic storage devices such as random access memory (RAM), or read on.y memory 
(ROM); or any other physical device or medium employed to store a computer program 

[0017] F,g. 1 A illustrates a system 10 that is useful in practicing the present invention. The system 10 includes a 
Z^^T*!?! 2 f ,r taini " 9 3 Central Passing «* (CPU) that can execute a set of predefined steps in 
carrying out the method of the present invention. A digital storage media 20 is also provided in connection with PC 12 
for storing (digital images. The digital storage media 20 can include different types of devices, such as RAM ROM 

im™ 1 °JZ T'^ di9ita ' St ° ra9e mSdia 20 030 a ' S0 be USed t0 ,oca,, y store the venerated enhanced 

Zfces o, I?' 0 "' 9 SUCh ** * SCanner 28 and a di9 * al camera 30 ' w "ich are addttiona. 

sources of digital images, can also be provided to the computer 12. However, it is to be understood that the digital 

Z?of Z^fTTT"! any S ° UrCe - A ^ intSraCtS Wfth the COmpUter 1 2 via in P ut devices *■ as a mouse 
and/or keyboard, and a display monitor 50 that is connected to the computer 12. The system 10 may also contain a 

device such as a printer 34 for locally outputting the images. Typically, the above components wou.d reside on n the 
sense of being directly connected to, the PC 12. ' 

EU3 J^T^' ab ° V ! com P° nents d0 not have to all reside on the host computer 12 but can reside on a 
server 62 that can be connected to a client PC 12 via a communication network 70. The server may also contain a 
central processing unit (CPU) that can execute a set of predefined steps in carrying out the method of the present 

imaoisto h S6 TT may "'I 0 bS C ° nnSCted t0 3 St0ra 9 6 media 65 and one P^ers 60. Th* can'enab e 

•mages to be remotely acquired, stored and printed via the communication network 70 using the storage media 65 and 
pnnter 60 connected to the server 62. The software for carrying out the present invention is typically stored on storaae 
media 20. Attentively, this software can be downloaded from the server via the commutation network The 
software for carrying out the present invention can be executed either on the client using the CPU contained in the PC 
12 or on the server side using the CPU contained in the server 62. The communication network 70 may comprise a 
pnvate network, such as a local area network (LAN), or a public network, such as the Internet that can be accessed 
by an mdividua using an Internet Service Provider (ISP). As is customary in such networks, the remote network service 

so r P 00 V 9, er r ? r k aCC6SSed ^ 3 CUSt ° mer 08109 3 retail ki ° Sk ° r any ° ther a PP'°P"ate communication device. 
[0019] F,g. 1 B shows an example of Graphic User Interface (GUI) for the software that carries out the present 
invention; the software runs locally on the system 10 or remotely on the server 62, and produces a GUI screen 78 as 
shown ,n Fig. 1E* The user launches the software and downloads an image to be enhanced. When the imaqe is 
downloaded, the Graphic User Interface screen 78 is displayed on the display 50. An image 80 on the left of the GUI 

55 thTrinJ 18 °T a ' downloaded ima 9 e with the face to be enhanced by the method of the present invention. On 
the nght an image 82 with the enhanced face is presented. In one embodiment, when the image is downloaded the 
f 6S ,° f f the ,aCe 80 10 be enha " ced - -sponse, as will be described in detail later, the system 
automatically finds facal feature points and segments the face into different features (e.g., eyes, eyebrows etc ) and 
a neck region. The system sets up default parameters and applies all enhancement filters in a predefined order to the 
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original image. 

[0020] The resulting image 82 is displayed on the right side of the GUI screen 78. The sliders 90, 92, 94, 96 and 98 
allow the user to interactively change parameters of different enhancement filters. The initial positions of the sliders 
correspond to the default val ues set up automatically by the system . The main appearance enhancer slider 90 combines 

5 all component enhancement sliders. The component sliders include a texture enhancer slider 92, a skin enhancer 
slider 94, an eye enhancer slider 96 and a teeth enhancer slider 98. The texture enhancement slider 92 controls 
parameters of the texture enhancement filter. The skin enhancer slider 94 controls parameters of the skin tone en- 
hancement filter. The eye enhancer slider 96 and the teeth enhancer slider 98 control parameters of the eye and teeth 
whitening filters, respectively. All the enhancement filters are described in detail in the following sections. The minimum 

10 and maximum for all sliders are set up to "no enhancement" (e.g., at the left extreme of each slider) and to "maximum 
enhancement" (e.g., at the right extreme of each slider), respectively. 

[0021] The user can control the level and look of facial enhancement by using the one global appearance enhancer 
slider 90 or the separate component sliders 92 - 98. Whenever the user changes a position of the main appearance 
enhancer slider 90, the system maps the position of the slider into appropriate parameter values of the enhancement 
15 filters and applies all the enhancement filters in the predefined order to the original image. The enhanced image 82 is 
then displayed on the right side of the GUI screen 78. Whenever the user changes one of the component enhancer 
sliders 92 - 98, the system applies all enhancement filters to the original image in the predefined order based on the 
positions of each component enhancer slider. The enhanced image 82 is then displayed on the right side of the GUI 
screen 78. Part of the GUI design in the preferred embodiment is the option of modifying facial feature points and a 
20 neck region outline. When the user selects that option from the menu "Edit" pulled down from the top bar of the GUI 
screen 78, the facial feature points and neck region outline points are overlaid on the original image 80 and the user 
can modify location of the displayed points by using the user input device 40, such as a pointing device. The tool bar 
84 contains specific tools the user can use to interact with and modify the displayed images. For example a tool for 
zooming in and out, a tool for editing feature points, a blending tool to locally blend the original image with the enhanced 
25 , image, a tool for spatially modifying the results of an enhancement filter, etc. 

[0022] Fig. 2A is a schematic flowchart illustrating one embodiment of the method of enhancing a portrait image 
according to the present invention. After initiating the process in step 200, a digital image is acquired by the system . ^ 

and then displayed on the display monitor 50 in an acquisition and display step 205. In the present invention, a digital ^ 
image refers not only to images obtained from photographs, but to digital images obtained without limitation from any ^ 
30 .source, for example, from a digital camera, scanning of a hardcopy image, or electronically from another source. In a ^ 
locate step 210, the individual faces in the image are detected and the location of the facial feature points on each ,^ 
-face are Identified. The process of locating the faces and their associated feature points can be performed manually 
by the user, or semi-automatically or automatically using image processing techniques. The locations of the facial ^ 
feature points are used to identify and segment different regions of the face (such as skin, eyes, nose, mouth, hair, >^ 
35 etc.) and the neck region. In a display step 215, the facial feature points are overlaid on the image and displayed on ^ x 
the display monitor 50. Optionally, feature points outlining the neck region are displayed as well. If the facial feature ^ 
points were determined automatically or semi-automatically, the user at a decision step 220 has an opportunity to ^ 
decide if the feature points need to be adjusted. If the user decides that the feature points need to be adjusted, at a 
modify step 225 the user can adjust the facial feature points using the input device 40. Optionally, the user can adjust 
40 feature points outlining the neck region as well. 

[0023] At a default setup step 230 the system uses the location of the facial feature points to identify and segment 
different regions of the face, such as skin, eyes, nose, mouth : hair, facial hair, etc., and determine the appropriate 
default parameters for the enhancement filters. In the preferred embodiment the neck region is automatically segment- 
ed based on location of facial feature points. The method of segmenting a neck region will be described in detail later. 
45 Accordingly, in step 230 the system determines the appropriate default parameters for enhancement filters for the neck 
region as well. Optionally, at this stage the gender and age of the face can be determined manually or automatically 
using gender and age classification algorithms. An example of automatic gender classification is described in B. 
Moghaddam and M.H. Yang, "Gender Classification with Support Vector Machines" in Proa of 4 th IEEE Int'l Conf. On 
Face & Gesture Recognition, March 2000, which is incorporated herein by reference. Typical algorithms for age clas- 
50 sification operate by performing an analysis of a digital image of a subject's face for particular detailed facial features 
such as facial feature ratios and wrinkle analysis. An automatic age classifier algorithm useful for the practice of the 
present invention is disclosed in US Patent No. 5,781 ,650 to Lobo and Kwon, which is incorporated herein by reference. 
Gender and age classification can be used to decide on which enhancement filters should be executed along with 
gender specific parameters. For example, as a default a larger amount of texture and skin enhancement can be applied 
55 to female faces than to male faces. 

[0024] The system then executes the enhancement filters in an execution step 235 using the appropriately chosen 
parameters. Alternatively, as shown by broken line, the user at a user selection step 240 can decide which enhancement 
filters should be executed along with the appropriate parameters for the enhancement filters. After the system finishes 
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™™ e 9 „ IZ ' hancement fitters in the st «P 235 the enhanced image is displayed in a display 245 on the display 
montor SO^At this point the user can decide if the enhanced image is acceptable, .f the image is not acceptable the 
S Ch00se e,th ( er u to cha "9* the enhancement filters and/or their parameters in the user selectSo 24^ or 
adjust the tocat.cn of the facial feature points in the modify step 225. In one embodiment, the usTcan choose £ 
ESSE? T dWerem enhancement fi,tere 10 see th. effect a given enhancement fitter has on thetaTe n 
th.sembodiment.theuserwillcycle through steps 240, 235, 245, 250 changing the enhancement fitters a^^ 
™££X"? fe ° bt * ined - ,fthe -^nced image is deemed acceptab,e in a decSo sTe^ So Te use" 
can chose either to locally pnnt in a local print step 255 on the printer 34 or locally archive in a local archive sVeo 260 
on storage media 20. Alternatively, if the system is connected to a server 62 via a communicattonSn 7 J the user can 
also choose erther to remotely print in a remote print step 265 on the printer 60 or remotely archive in a rel^e archive 

the ofesen^? 96 ^ ^ aCti0n Cn0Sen and «""P'^ the sysZqueS ^ 

slpTos for thl ne^m? 96 T T *"> °" reSP ° nSe> back t0 the ac « uire and ^p.ay 

step 205 for the next image or ends the process (ending step 280). 

[0025] Fig. 2B shows an alternative embodiment of enhancing a portrait image according to the present invention 

embodrrH 0 "^ 6 hi9h ' y aUt ° mated ^ 6ffiCient pr ° dUCti0n at 3 comme " ia ' Photographic 
embodiment descnbes an automatic batch process for running a large number of images through the portrattTnhancT 
ment system without the necessity of user intervention. At an acquisition step 282, an ZaTisZuZTnT^ 
computer file) of images that have been selected to be enhanced. At a face detection and3t£n step he\ac£ 

aU h t0matiCa,,y delGCled and ,he ' 0CatiOnS ° f the facial fealure P° inls °" * a <* 'ace are automat Sy 
Tll ^l SeleCti ° n St6P 286 ' thB SySt6m *"*«*« the face and determines the apTroS 

andTa^^ 

he^einbeforir **" aUt ° matiCa " y USi " 9 9ender and a ° e ^ification algorithms (as des'cled 

[0026] Alternatively, the age and gender of the faces in the image can be supplied to the system via metadata as- 

W,th ,ma9e ' ?*■ thS 396 and 9ender can be s "PP |ied f ™ - customer.supp.ied IhoJSSS^L 
orby otherw.se querying the user. The default fitters and parameters can a.so be supp.ied as image dependTntmetSa 
or as an image .ndependent setting before the process of enhancement is imp.emented. The systeTthen executes 
hee nS e mb , eo enhancementfitters in an enhancement execution step 288 using the appropriately ch^ 

Zsl^and^^u^ 

steps 286. 288. and 290 until all faces in the image are enhanced. At a storage step 292 the enhanced imaae is stored 

24 e )^.l e l anCin h 9 ""r 965 18 C ° ntinUed Unt " a " thS inpUt ima96S haVe been enhancedtmTge'q e^Tep 
^ffif/w «n m *9 es havebeen processed (ending step 296), the enhanced images maybe appliedto the utifeation 
stages 255- 270 shown ,n Figure 2A. Optionally, after all the images have been processed (ending step 296) me 
enhanced images can then be brought into the user interactive system, such as described in Fig 2A to be ^checked 
and .f needed modified before the utilization stage (steps 255 - 270 in Figure 2A) 

52?.- R .k 3 iS f f,0Wchart forthe facial detec «°n and point location step 210 of Figs. 2A and 2B, showing the process 
of ocating the salient facia, feature points on the faces present in the image according to the present Zen ton Aface 
detect.cn module 31 0 is applied to the digital image to mark the tocations and rough sizes of all SnZS^J^St 
in the .mage, and a facial feature detector 31 5 locates the salient facial feature points on the detecteoT^ 
mSdolo S a T e P ,;h:t b h e Pe H 0m,ed manUa " y by thS US6r ° r aL — <*"V "sing'one of the nunJi^iSSj 
thk 9 „„ ST VS reC ° rded ^ thS 8CademiC Hterature - A P referred automatic face detection memod for 

this app . cation consists of methods described in Heniy Schneiderman, A Statistical Model for 3D g £j ?DeZton 

K CeS f ndGare - Pn °- Th6SiS ' ROb ° ,iCS ,nstitUte " Came 9 ie M ellon University, May 2000, wSilS 
t?T k onh 2 en . Ce ' Alternative, » if a manual a PP roa <* * to detect faces a preferred method Ts for he user 

l Zl?L^TT 9 d f ecti0n of one or more faces . the image coordinates and estimated size of each face are in turn 
provided tothefacialfeature detector315, which has the taskof locating the salient facia, feature points onThe ejected 
faces n the preferred embodiment, an active shape model is used as the facia, feature detectorTe ac^e lhaoe 
mode is desenbed in A. Lanitis, C.J. Taylor, andT.F. Cootes, -Automatic interpretation and codmg ot face imaqes us^o 
fiex.b.e models," /^E Trans, on PAMI, Vol. 19, No. 7, pp 743-756, 1997. which is incorporated J^SSZ? 
The det ected feature po.nts are used to identtty, outline, and segment different regions of the face, such LTeTacial 
skin region, eyes nose mouth, eyebrows, hair, facial hair, etc., and a neck region The detected regions are derSd 
mJ£ T Tl : d ' ng T arV maSkS - The maSkS ° f the re9i0ns are then s P atial, V fathered to genera* a feha 
enttln L rS ^ USed in StSP 235 and 288 l ° b,6nd ,hS resu,tS of the enhancement filter (e.g toTre 
t~r»£ T) W,t ^ heo "9' na,ima g e Bering binary masksandapplyingtheresultingalphamasksinblenXq 

operation ensure smooth transition between regions that have and have not been enhanced. To generate alpha masks 
the binary masks are feathered by blurring the binary masks with a blurring function where the blur radiuf is chosen 
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based upon the size of the face. The binary masks are used to determine where to spatially apply the enhancement 
filters as shown in Fig. 2A and Fig. 2B. 

[0029] Referring to Fig. 4, there is shown an visual example of the location of salient feature points 420 on a detected 
face 410. Typically these facial feature points are located either manually or automatically using image processing 
5 techniques. 

[0030] In many images it is critical to apply the skin enhancement filters not only to the face region but also to the 
neck region. In a presently preferred embodiment, the neck region is determined by combining a modified generic neck 
shape model with a skin color classifier. The flow chart for the method of determining neck region is shown in Fig. 5A. 
In a generic mapping step 550, a generic probability map for a neck region is created. A generic neck probability map 

10 is based upon a priori knowledge of the shape of the neck. In the preferred embodiment, a generic probability map is 
created by averaging a sample population of normalized images that represent an anticipated population of images to 
be enhanced (e.g. portrait type images). The sample images are normalized by scaling each image to the same pre- 
defined location of the eyes. In each sample image, a neck region is outlined manually. The final neck probability for 
each pixel is an average sum of a scoring function equal to 1 if a given pixel is a part of neck region and 0 otherwise 

is for all sample images. A generic neck probability map can be created using heuristic approximation. If a gender of a 
person in the image is known, a different probability map is used for men and women in the preferred embodiment. 
Usually, a skin region is more visible in portraits of women than of men. In a scaling step 554, a generic neck probability 
map is scaled to fit individual feature points of the chin line and the face outline. 

[0031] In a skin color classification step 556, a neck probability map is created based on color segmentation. A 
20 supervised pixel-based color classifier is employed to mark all pixels that are within a specified distance of skin color. 
The pixel-based color classifier, which is a conditional probability function of pixel color C belonging to skin, is modeled 
as a Gaussian, 

p(C\skin) = ^expf-^C-H^^EjLCC-li^)] (EQ. 0 

: 2n\L tUi y 2 L 2 J 

30 where mean vector u. and the covariance matrix I are estimated from the defined skin region. The vector C corresponds 

to the pixel's red (R), green (G), and blue (B) signal. The above approach is also applicable when C is represented in * 
other color spaces, e.g., CIELAB, YU V, HSV, etc. A subset of the facial skin region is used to determine a conditional 
skin probability distribution. In a presently preferred embodiment the skin region above the eyes and skin regions ^ 
where . facial hair is detected are excluded from use in estimating the mean vector \i and the covariance matrix I in & 

35 equation EQ. 1 . The neck probability is defined according to equation EQ. 1 for all pixels outside the face region and 
below the chin line, and is equal to 0 otherwise. 

[0032] In a final map generation step 558, the final probability map is created by combining the scaled generic neck ^ 
probability map and the skin color based probability map. In the preferred embodiment, two probability maps are ar- 
ithmetically multiplied. The resulting neck probability map is used as an alpha channel to determine how to blend the 
40 results of the enhancement filters with the original image. The binary neck region mask is created by thresholding the 
final neck probability map. If the probability is greater than 0 for a given pixel, the corresponding mask value is equal 
to 1 thereby indicating the neck region, otherwise the mask value is equal to 0 thereby indicating a non-neck region. 
The binary neck region mask is used to determine whereto apply the enhancement filters. 

[0033] Fig. 5B - 5D demonstrate visually the process of creating the final neck probability map. Referring to Fig. 5B, 
45 a scaled generic neck probability map is shown. A generic probability map is scaled to match the individual feature 
points outlining the face. A generic neck probability map is based on a priori knowledge of neck location relative to the 
outline of the face as described in the previous section. The generic neck probability shown in Fig. 5B is one possible 
example based on heuristic rules rather than statistical analysis recommended in the previous section. It serves the 
purpose of demonstrating qualitative characteristics of the process. A central region 570 within the generic mask has 
so high values (e.g. greater than 0.5 and less than or equal to 1 ) corresponding to the high probability of the neck region. 
A border region 572 has lower values (e.g. greater than 0 less than 0.5) corresponding to the lower probability of the 
neck region. The probability of the neck region tapers off to a value of 0 outside the region 572. In general, the probability 
decreases in a continuous manner from the center of the region 570 to the edges of the region 572 in a horizontal 
direction and from top to bottom in a vertical direction. The central region of the mask that is right below the chin line 
55 has the largest probability associated with it. 

[0034] An outline 574 of the neck probability map determined by a skin color classifier is shown in Fig. 5C. The skin 
color based probability is calculated according to the equation EQ. 1 as described in the previous section. The prob- 
ability values outside the outlined region 574 are equal to 0. The probability values within the region 574 are greater 
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n~ k °" y ,f qU T" ° 1 ThS ^ n6Ck P robabilitv ma P s: ,h e scaled generic neck probability map and the 

neck probability map based on skin color classification are combined by arithmetic multiplication of the two. The outline 

rlln ™ ? 9 i Tt L Pr0babi,ity map is shown in R 9- 5D - antral region 576 corresponds to the high probability 

5 m^^T I f K Pr0babiHty re9i ° n 574 The re9i0n 578 ^responds to the .ow probabSity region 

s 572 cropped by the skin color probability region 574. 

»n??L 0nCe thG 5? al featUre P ° intS 3nd n6Ck r69i0n hav£i b<?en ,ocated ' ensemble (i.e.. two or more) of appear- 
enhL^Z^rif S f 66 aPP ' ied t0 thS ,aCe and n6Ck re9ions in the imaae - Refemn 9 10 6, several different 
!" h n " t k S 'I™"" ,n ,he PreferrSd embodiment - «ie fo"owing enhancement filters are implemented a 
skm texture enhancement filter 610, a skin tone enhancement filter 620, a teeth enhancement filter 630 an eye en 
to hancement f, ter 640, and a facial feature shape enhancement filter 650. All these filters are described in detain The 
followmg sections of the specification. The enhancement filters can be applied in any order. In one embod^en the 
user can select any of the enhancement filters in any order he/she wants to apply them to the faces and ZTreaZl 

ssrs ^rc""*' ,he system sets a defau,t ° rder ° f app -^ enn — 

' 5 ESS " f Sr e,eCtS t0 mn de,aUlt SSttingS ' then a " ennanc oment filters are applied in the default order with the 
default parameters and the enhanced image is displayed on the monitor. The defauE order of enhancemen tiers is 

e he older TtT, enhancament tha tone enhancement fitter, the teeth and eye whitening Z? ( in 

m the same order as the default order of applying filters to the original image. 



55 



Texture enhancing filter. 

ES? , J?" ^ teXtUre enhancin 9 ,ilter is t0 smooth the 'ocal texture of the skin, remove blemishes, dark 
ZT h J« < °- , rem ° Ve ° r l£SSen tHe GXtent and deepn6SS ° f Wnnkles Referrina t0 Fi 9- 7A, there is shown a 
ITT , I 3 Z 6 enhanCln9 filtSr aCCOrdi " 9 l ° thS Present invention - ,n a ski " mature generation step 710 
the skm features that are within the skin region delineated by the facial feature points are detected. Then the detected 

such as T 3 h m °TT " 3 featUre m ° dif ' Cati0n S,6P 720 Skin featUre ma P Contains feat -os that we wish >U mod * 
such as blemishes, dark spots, wrinkles, etc. ■■«uny, 

[0038] In the prior art, color analysis is used to locate specific skin defects. Also in the prior art the detection of skin 

TZ^nT^f ^ r^*,"? hiSt ° 9ram ° f ,hS C ° l0r Si9 " als associat ed with the pixels contained within a finite 
2-d.mensional window that is slid over the skin region. A region is considered to contain a skin defect if its associated 

tZZ ?r J < ,m T 9 S , yStem b6 C0 '° r Ca,ibrat6d and/ ° r the image t0 be ana| y zed be oorrecTly color balanced 
me im!^. "Z 9 C ° r ,nf0rmat,on is enhanced when oontrolled lighting and/or coiored fitters are used to capture 
the .mages. In addrt.on, accurate color reproduction is needed if the detected defects are going to be further classified 
as blemishes, brown spots, etc based on the color of the defect. ciassmea 
furtfhlh l nnht h0t09 ^ aP , hS ; t,ie skin , im P erfecti ° ns such a * wrinkles, blemishes, etc manifest themselves as (are really 
nfL? 9 f ad ° WS PlaC6S WhGre *"* Sh0Uld n0t be 70(5 skin imperfections thus correspond to local 

nan- l7*T ^ T™, * COrres P° nd t0 P eaks and tha abad °ws correspond to valleys in the E£! 

ZZrfZT a de P art " ra ,rom the P rior art > a " d wording to a preferred embodiment of the current invention, skin 

SSL? IZ ! . I SPP¥mg Peak and Va,,ey deteCt0rS t0 the ,uminance **** f °™ ed *™ «» component 
RGB signals. Valley/peak detectors are second difference operators. The luminance is defined as a weighted linear 
combination of the red R, green G, and blue B signals, as follows, we.gntea linear 

L = k1R + k2G + k3B ( EQ 2 ) 

T-V va k 'r ; ep ;r F nt the r i9 T- ,? p : esent,y pre,erred choice ° f wei9n,s for 9eneratin 9 the ■«*»»« **** * 

Kl - k2 - k3 = 0.333. Examples of valley/peak operators can be found in D.E. Pearson and J.A. Robinson "Visual 
Communication at Very Low Data Rates," Proceedings of the IEEE, Vol. 73 No 4 April 1 985 ' 
[0040] A presently preferred peak/valley operator is a series of four directional operators of the type employed bv 

Fia a ?B n w e e 8 ,° P T 0r V ' Wh J Ch iS aPP ' ied t0 ,he midd ' e Pixe ' m the 5x5 neighborhood or pixels'shown in 
F.g. 7B, where the pixel locations are designated a through y, is defined by equation EQ. 3, as follows, 

V=1+k + p+j+o + t- 2(h + m+r) (EQ. 3) 

This operator is applied at each pixel location in the delineated skin region. Similarly, a horizontal operator H of the 



BNSOCCID: <EP 1372t09A2J_> 



8 



EP 1 372 109 A2 



form shown in equation EQ. 4 and a pair of right and left diagonal operators DL and DR as shown in equations EQ. 5 
and EQ. 6. Respectively, are applied at each pixel location in the delineated skin region, as follows, 



5 



10 



H = b+c + d + v+w+x~2(l + m + n) 
DR = c + g + k + o + s + w- 2(i+m+q) 
DL = c + i + o + k + q + w- 2(g + m + s) 



(EQ. 4) 
(EQ.5) 
(EQ. 6) 



These operators not only detect valleys/peaks, but they also have a secondary response to the feet/shoulder of edges. 
Thus they are referred to as valedge/peakedge detectors. Valedge features correspond to positive values of the op- 
ts erator*s output whereas peakedge features correspond to negative values. A valley/peak detector that is sensitive only 
to valleys/peaks is obtained by applying logical conditions to operators. For vertical valleys the logical valley detector 
correspondence is given by: 

\Uf+k+p)>(h+m + r) and (j + o + t) > (h + m + r) 

20 

then V=(f+ k+ p + j+ o+ f ) - 2{h + m + r) 

elseV=0 (EQ. 7) 

25 

For vertical peaks the logical peak detector correspondence is given by: 

if (f+ k +p) < (h + m + r) and (j + o + t) < (h + m + r) 
30 . _ then V= (f+ k +p +j +o+t)-2(h + m+r) 

elseV=0 (EQ.8) 

35 Logical detectors for a horizontal and diagonal valleys/peaks have similar form. Both valley/peak and valedge/peakedge 
operators are effective for generating the skin feature map according to the present invention. From this point on we 
use the term valley/peak operators to refer both to valley/peak and valedge/peakedge operators. 
[0041] Prior to applying the oriented valley/peak filters to the image, the effects of noise in the image are suppressed 
by applying a noise reduction filter. Appropriate noise filters are low pass filters, median filters, and other linear and 

40 non-linear filters commonly employed to reduce noise in digital images. 

[0042] The oriented valley/peak images generated by the operators are thresholded so as to retain only (strong) 
relevant skin features. For valley feature maps, pixels with values less than a specified threshold T v are set to zero. 
For peak feature maps, pixels with values greater than a specified threshold T p are set to zero. The threshold can be 
either a fixed global threshold or an adaptive threshold. A presently preferred threshold method is to use an adaptive 

45 threshold whose value is given by equation EQ. 9. 



T^L avg (EQ.9) 

so where p is a constant and L avg is the local average luminance about the pixel. Different values of p can be used for 
the vertical, horizontal, and diagonal components. The local average luminance L WQ may be the value of the pixel itself 
or the average luminance of a neighborhood of pixels, for example a 3x3 neighborhood of pixels. 
[0043] A presently preferred step is to generate a combined skin feature map F by combining the individual oriented 
feature maps. 

55 

F = max{H, V, DR, DL} (EQ. 10) 
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Additionally, each pixel in Fcan be labeled according to which oriented fitter it originated from. The oriented label data 

of"f oil 8 , T ! ? riented featUre mapS C3n be USe,ul in removin 9 ,eatures such as inkles where features 
of a particular orientation are preferentially removed. 

lllVl t J h t V aS i°\ Xh l Ski " teXtUre enhancin 9 filter is t0 smooth the local texture of the skin. The spatial size of the 

i? , 6 . anCi " 9 fi,ter Sm ° 0,heS iS 3 f UnC,ion of the size of the 'ace. Thus the spatial size o he 
valley/peak filters ,s adaptive* determined based upon the size of the face. Specifically, the separaL between the 

operator" " iZS ° f thS Va,ley/Peak filter ° ne generic ^P^entation for the vertical vaTy/peak 



10 



1 h 

y ) = — X 7 < x - w > v + 0 - 2/(x, y + o + /(* + *y> + o (eq. i i) 

15 

» HL SSLCr 1 is °" op " ! va " ey/p * ak * m " fe 9,vsn b> sub " K * n9 b ""™ lu "" n ""» 
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F(x,y)=/(x,y)-/ 6 (x,y) (EQ. 12) 

The blurred luminance image can be generated by applying a blur filter such as a Gaussian or a box filter to the 
lummanoo imago. The output of the val.oy/poak detector is thresho.ded as described above to geneL peal and 
va ley feature maps. The radius of the blur filter is chosen as a function of the face size (which I an example of a 
default parameter) and the size of the skin feature that one wishes to detect P 

ESS i J h r fe T 6 m ? I 3 ' 6 fUrthSr ref ' ned by 9r0Uping piXe ' S that are co "^ted to each other via connected com- 
conned S ? 9 T , C ° mp ° nent ,abelin 9 scans an ima 9 e and 9™ups its pixels into components based on pixel 
nZSl l ' h ^ 8 C r neCtSd C0m P° nent are in some way ^nnected with each other. Once all groups 

corresponds to a skin feature. Each skin feature is characterized and classified by its feature-based characteristics 
such as s*e, shape, and location in the skin region. The size corresponds to the number of pixeStn the SmponS 

Stlf" eXa 7' e k! 8 Parameter> - ShaPG in,0matiOn iS USeful in deli - ati "9 -nkle el" res from othe 

sk,n features such as blemishes. In addition the features delineated in the skin feature map can be further classifkfd 

2 1"! S° r ,n, t T ati ° n aSS ° Ciated W " h the PiX6,S that h3Ve been identified as skin OnceS skin ffatere 

D 9 , erat6d th6y 3re USGd l ° S6l6Ct the skin features that are 9° in 9 t0 be modified in step 720 

9 ? R9 -' 8 t L th6re iS Sh ° Wn 3 f ' OW Ch3rt illustratin 9 a preferred embodiment of selecting and modifying 
skin features according to the present invention. In a feature selection step 810. features to be modified are selected 
based on their characteristics. As mentioned above, the feature characteristics correspond to the sS sha^nd 
color and are selected based upon the location of these features in the skin region. In a dilation step 820 the selected 
skin features are then (adaptively) dilated and then modified in a filtering step 830. The effect of dilat on on a sSn 

zUntelll ^ T f e ° f f6atUre The am ° Unt ° f di ' ati0n Can be either a fixed a — » fo all defected 

to^TZ * P t,VG am ° Unt baS6d UP ° n the ^aracteristics of the skin feature. In a presently preferred em- 

T»Zt? T\ U ? S adaptiV6ly dila,ed baSSd Up ° n their size and the size of face (whic£ °s an example 
of a default parameter). Larger skin features are dilated more than smaller skin features. The process of dilation can 
be earned out using standard morphological operators. 

of 0 ™ aaT^n'iSn" e h n , han h Cement '« > ot ' tained ^ sequentially modifying skin features by cycling through the steps 

si! Elf \ ZZL r n9m9 ° f SKin featUre 10 be m ° dified ' albeit 311 th e desired skin features can be 

selected and modrf.ed in one cycle through the steps 810, 820, and 830. In a presently preferred embodiment the vallev 

andmo a S "T' ^T* StepS 81 ° " *»• a " d the " the ^^uai peak skin to«uZS££2 

whose 2£ 1 * hr ° U9h StePS 810 " 830 ° rder t0 P reserve the s "n texture, only skin features 

m/«l f!f b ^t en f a SPeC ' f,ed minimUm a0d 8 maXimUm Size are modified " Alternatively, in order to preserve 

m^lm fl nr' ' 6atU : eS WhOSS SiZ6S ' arger than 3 Sp6Cified minimum size are mo ^ied. In addWon the 
mTiimum and maximum size of the skin features to be modified directly scale with the size of the face 

[0049] In step 830, a pinwheel filter is applied to each pixel of dilated features. Pixels of dilated features are referred 



BNSOOCIO <EP 1372109 A2_l_> 



10 



EP 1 372109 A2 

to as feature pixels. All other remaining pixels are referred to as non-feature pixels. In the embodiment of the present 
invention, feature pixels are defined by a binary mask, where a value of 0 corresponds to feature pixels and a value 
of 1 corresponds to non-feature pixels. The pinwheel filter interpolates new values of a given feature pixel by using 
neighboring non-feature pixels aligned in line segments centered at the feature pixel. The pinwheel filter is described 

5 in detail in commonly-assigned U.S. Patent No. 6,104,839 "Method and Apparatus for Correcting Pixel Values in a 
Digital lmage M : which issued August 1 5, 2000 in the names of David R. Cok et al, and which is incorporated herein by 
reference. A brief description of the pinwheel filter is included here for clarity and to specify some modifications to the 
original algorithm as described in the patent by Cok et al. Referring to Fig. 9, the SET of four line segments 930 is 
graphically demonstrated (vertical V, horizontal H, two diagonal line segments D1 and D2) for the selected feature pixel 

10 940. The four line segments are spaced at 45° degree increments. The dilated features 920 are represented in gray 
(shaded) color. The line 910 represents a face boundary. Each line segment in the SET is composed of both feature 
and non-feature pixels on both sides of the pixel 940. The non-feature pixels and feature pixels in the line segment 
are pixels local to the selected feature pixel 940 in a given direction defined by the line segment. The method of creating 
the SET of line segments is described in detail later. 

15 [0050] Referring to Figs. 10A and 10B, the method for calculating new values for each feature pixel in the image by 
applying the pinwheel filter is shown. In step 1004, the number of line segments NL the maximum number of pixels 
on one side of the line segment MAX_NP1 and the minimum number of pixels on one side of the line segment M IN_NP 
1 are set. These parameters will be explained in detail in reference to Fig. 12. The maximum number of pixels on one 
side of the line segment MAX_NP1 and the minimum number of pixels on one side of the line segment MIN_NP 1 are 

20 set based on the size of the face (which is an example of a default parameter). The larger the size of the face, the 
larger the values of MAX_NP1 and MIN_NP1 . The dependence of these two parameters upon the size of the face is 
unique to the present invention compared to the method described by Cok et al. In step 1006, the first feature pixel 
PSEL is selected. In step 1008, the SET of NL line segments is created through the pixel PSEL. In the preferred 
embodiment, the number of line segments NL in SET is equal to 4, corresponding to vertical, horizontal and two diagonal 

25 line segments. 

[0051 ] A method of creating line segments is shown in Fig. 1 2. In step 1 1 10 of Figure 1 2, the first segment is selected, 
e.g., the vertical line segment (V). In step 1120, the direction of one side of the line segment is selected. This is a ^ 
direction in which pixels are being added to the line segment starting at pixel PSEL. For each segment, two directions .« 
are defined that correspond to two sides of the line segment centered at pixel PSEL. In step 1130, feature pixels are 
30 being added to the line segment until a first non-feature pixel is reached along a given direction. In step 1140, the first ^ 
non-feature pixel is added to the line segment and the neighboring non-feature pixels in a given direction are being * 
added to the line segment until one of the following conditions is met: 

maximum number of non-feature points on one side MAXJMP1 is reached, 
35 - face boundary or face feature boundary is reached, ^ 
new feature pixel is reached. - - > 

• * 

The maximum number of non-feature points on one side of the line segment MAX_NP1 is set in step 1004 (Figure 1 A) 
based on the face size. The line segments must not cross the face boundary or boundaries of the regions excluded 

40 from the texture enhancement process (like eye regions, mouth regions, etc.). The above constraining condition for 
creating valid line segments is unique to the present invention compared to the method described by Cok et al. When 
adding pixels to one side of the line segment is completed, then the direction is set for the second side of the line 
segment in step 1160, and the process is repeated for the second side of line segment in steps 1130 -1140. When two 
sides are completed (step 1150), then the next line segment from the SET is selected (step 1180) and the process is 

45 repeated in steps 1120 -1160. When all NL line segments In SET are created (step 1170) the process is completed 
(step 1190). 

[0052] Referring back to Fig. 1 0A, after creating the SET of NL line segments in step 1008 as described above, the 
line segments having less than MINJMP1 of non-feature points on at least one side are removed from the SET (step 
1010). If no remaining valid line segments are left in the SET (step 1014), then pixel values are not changed for the 

so pixel PSEL (step 1052) and a new feature pixel PSEL is selected (step 1046 and 1048) and the process is repeated 
(starting with step 1008). If the number of remaining valid line segments N in SET is greater than 0 (step 1014), then 
for each remaining valid line segment a linear fit is calculated (step 1016) for the non-feature pixels in the valid line 
segment for each channel. In step 1018, mean square fitting error is calculated for each valid line segment for non- 
feature pixels for each channel based on the linear fit computed in the previous step 1016. In step 1020, total mean 

55 square fit error is calculated for each valid line segment as an average of mean square fitting errors for all channels 
for a given valid line segment according to the following equation. 
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1 K 

MS %> = ^ MS£ ** where nBl N (EQ. 13) 

PSEL _ OUT,, <,PSEL k U1-*>PSEL_IN k (EQ. 14) 

new values PSEL k for each channel k are equal to averaged values of PSEL < „ and PSPi <«t on i<Z ™T 

rf the number of va.id line segments N in SET is equal to 3 (step lO^t^^^SreE^^^T; 

SpT™ s:^: * L 2k ca h ,cu, r s rthe vaiid ,ine ^ ■ ^ ^sj^ k 



55 = 1 ~ H — where n = 1 ,..,N k = 1 ,. . .,K (EQ. 15) 
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Next, new values PSEL k are calculated for each channel as a weighted sum of PSEL n k values determined for each 
valid line segment n and for each channel k according to the equation EQ. 16. 



PSELk =Ji*— 7j (EQ. 16) 

5X, 

10 

The final pixel values of the enhanced image are calculated by blending new pixel values PSEL k for each channel k 
with the original pixel values according to alpha masks (step 1024) generated in the previous sections. The blending 
operation and alpha masks are unique to the present invention compared to the method described by Cok et al. 

15 [0056] The majority of the skin features that we wish to modify correspond to valley features i.e., a dark area sur- 
rounded by a light area. In most instances, skin features are going to be modified regardless of the color information 
associated with the pixels that have been identified as skin features, albeit there may be instances where an individual 
may not want a defining facial characteristic such as a beauty mark to be removed from the photograph of the individual . 
In these instances, the color information associated with the skin feature pixels can be used in determining the type 

20 of skin feature that should or should not be removed. An alternative approach is to build a tool into the Graphical User 
Interface that will allow the operator to undo an undesirable modification (e.g., the removal of a desirable beauty mark). 
In a preferred embodiment, the user chooses an undo/redo tool from the graphical list of tools 84, and moves the undo/ 
redo tool via pointing device 40 such as a mouse to the location of the skin feature in the original image 80 the user 
wishes to restore. Clicking on the skin feature in the original image 80, which is displayed on the left, causes the feature 

25 to be added back to the enhanced image 82, which is displayed on the right. Clicking again on the restored skin feature 
in the original image now causes the skin feature to be removed from the enhanced image. Thus, the undo/redo tool 
works, toggling back and forth 5 by either removing a skin feature from the enhanced image if it is present in the enhanced 
image or restores it to the enhanced image if it is not present in the enhanced image. In an alternative embodiment, 
the texture enhancer slider 92 can be set to no enhancement and the tool 84 can be used to allow the user to choose 

30 which skin features to remove. 

[0057] In another embodiment, the graphical user interface acquires and displays a digital image containing one or 
more faces. A skin feature map is generated by use of any of the aforementioned techniques to identify and map the 
skin features; the skin feature map therefore represents the skin features on the one or more faces in the digital image. 
The pointing device 40 is then used to point to a particular skin feature. In response to a point and click operation of 

35 the pointing device, the skin feature map is referenced as to the particular feature and the appropriate enhancement 
for that feature, provided by an appropriate enhancement filter, is initiated for the skin feature being pointed at. 

Skin tone enhancement filter. 

40 [0058] The task of the skin tone enhancing filter 620 (Fig. 6) is to improve the overall appearance of the skin tone. 
The skin tone filter evens out unbalanced skin tone and creates a more even colored skin tone. In addition, the skin 
tone filter is used to modify the color of the skin tone to produce a more desirable skin color representation. This 
corresponds to modifying both the luminance and chrominance of the skin to match individual and culture preferences 
of skin tone rendition. Moreover, skin tone problems tend to be on a larger spatial scale than texture. It is important 

45 not to even out the skin tone too much because faces without highlights or shadows are flat and tend to be uninteresting. 
[0059] Referring to Fig. 13, there is shown a flowchart for a presently preferred embodiment of a skin tone enhancing 
filter according to the present invention. The skin tone-enhancing filter adaptively compresses highlights and shadows 
by adaptively blending (in a blending step 1240) the input image /(x)with a blurred version t b (x) of the input image 
obtained from a blending step 1210, as follows. 

50 

O(ix) = <x(x)/ b (x) + (1- ot(x))/(x) (EQ- 17) 

where a(x) is a blending coefficient obtained from a coefficient generation step 1230 and x = (x, y) is the location of a 
55 pixel. The blending operation is applied to all the color channels. The blending coefficient is a function of the shadow/ 
highlight SH(x) strength image at x obtained from the shadow/peak generation step 1220, as follows. 
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a(x) = /tSH(x)J (EQ. 18) 

I?ri 6 n°JL Referrin 9 t0 1 4. there is shown a presently preferred embodiment of calculating the shadow/highlight 
strength .mage according to the present invention. The shadow/highlight strength image from step 1220 is generated 

hi ^T^nTlT™™ ima9S 1320 fr ° m 3 ' UminanCe 131 °- ^e luminance image SSESS 

aJeraoe ZSnl a. , T ' UminanCe M " bS 6ither a local avera 9 e ,uminan <* image or a global 

orSii IT tn f aVera9e ,UmmanCe ima9e Can 66 96nerated WWng a blur filter such as a Gaussian 

or box filter to the lummance .mage, whereas the global average luminance is calculated by determining the averaae 
um.nanceofthesl<.n W ithintheskinregionde.ineatedbythefeature points. The shadow/highlight st™S 

emboSftH ' 0Cal aV6rage ima9e t0 Ca ' CU,ate the ^hadow/higNight imag a S 

o Tthe vSnl?H 7 a ? US l ° ^ a ' CUlate the sha < J ow/high. i ght map should be larger than the blur radius or extent 

f P , I "'^ t6XtUre 6nhanCing mef - eithercase - the blur radius is dependent upon the 

size of the face (which is an example of a default parameter). 

[0061] In its simplest embodiment, the blending coefficient is a constant /[SH(x)J = «(x) = a which is an examole 
o a default parameter, and is not dependent upon the shadow/highlight strength image as shown in Fig 15A The 

«ir r T n ? ,0nS h f T Fig ' 1 5C are US6,Ul f ° r t0nin9 d0W " hi 9 h ' f 9 hts and hot s P°ts cauTed by lights 

and/or oily skin, while leaving shadow regions unchanged. Y 9 

f^n 21 th The | Sk i T fi " er Can a,S ° bC US6d l ° m ° dify ,he °° IOr rendilion of lhe skin - ln a Primly preferred embod- 

S^VL'^J^^ skin region can be modified c - by shifting - — and — - - 

C mod (x,y) = k(C(x,y)-C)+ C prefetred (x,Y) (EQ. 19) 

where the vector C(x, y) corresponds to the pixel's red (R), green (G), and blue (B) signal, C DretBmd is the preferred 
co.or vector, C is the current mean co.or vector, and k is a constant that can vary between 0 ano~ preyed 
'"preferred K*.y) can vary spatially depending upon the region of the skin to be modified 
30 asTo^ PrefSrred emb ° dim6nt f ° r V V hXe ™9 ° r darken the skin region is to modify the contrast of the skin region 

c ™od(*..K)= Cy(x,y) (EQ.20) 

" tTdT^nlng 8 ^ sSoT C ° rreSPOnd t0 ' i9htening Skin COl ° r 3nd Va,UGS ° f Y 9reat6r tha " ° ne co - s P°" d 
[0064] An alternative embodiment for lightening the skin region is given by 

40 r — 

mod 1 - y(1-C) (EQ.21) 

and for darkening the skin region is given by 

45 r* _ ■* 1 ~ C 

mod T"^C (EQ.22) 

where y varies between 0 and 1 . 

[0065] The above approach is also applicable when C is represented in other color spaces, e.g CIELAB YUV HSV 
etc. In addrt.cn these equation can be applied to all or a subset of the components of C. This is particularly true when 

nanc^avT " ^ **** C ° mp ° nentS 8re re,ated t0 the luminance < L *> ™<* <*romi- 

Teeth and eye enhancing filter. 

55 n°.« 6 i th^f T ^ t6et o T d enhancin 9 fitter < s > 630 and 64 ° (Rg- 6) is to increase the luminance and white- 

S 1 T J 6 emn9 1 ° Rg - 1 6 ' ther ° is Sh ° Wn a flowchart for a P resent| y Purred embodiment of a 

teeth and eye whitening filter according to the present invention. At an eye/teeth mask generation step 1510 the saltern 
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facial feature points 420 (Fig.4) are used to generate a mask that identifies pixels that correspond to the eye and teeth 
regions. Using the input image at a valley/peak map generation step 1520, the valley/peak map is generated using 
Eq. 12 where the radius of the blur is determined by the eye separation (which is an example of a default parameter). 
At a scaling factor generation step 1530, the valley/peak map and the color information are used to calculate scaling 
factors (which are examples of default parameters) for the luminance and chrominance values of the pixels within the 
eye and teeth masks. Then at a pixel modification step 1540, the scaling factors are applied to the luminance and 
chrominance values of the pixels within eye and teeth regions generating new luminance and chrominance values. 
[0067] In a presently preferred embodiment, the RGB values for pixels within the mask regions are converted to 
CIELAB (L*a*b*) space and the luminance and chrominance values are modified as follows, 



L*=L*.(UkP) 



(EQ.23) 



15 



a* = a*/(1+/cP) 



(EQ.24) 



(EQ.25) 



20 



where k \s the aforementioned default parameter and P is the probability that the pixel belongs either to the whites of 
the eyes or to a tooth. A presently preferred expression for the probability P is as follows, 



25 



1- 



0 



if-fi<F<0 
otherwise 



(EQ.26) 



30 where F is calculated using Eq. 12 with a blur radius chosen as a function of the face size and p is a threshold. 
Shape enhancement filter 

[0068] The task of the shape enhancing filter 650 (shown in Fig. 6) is to modify the shape of facial features such as 
35 the eyes, nose and mouth. The shapes of the facial feature are enhanced by modifying their shape using standard 
warping techniques. An example of a warping technique useful for practicing the current invention is given by T. Beier 
and S. Neely. Feature-Based Image Metamorphosis, Computer Graphics, 26(2): 35-42, New York, NY, July, 1992, 
Proceedings of SIGGRAPH '92, which is incorporated herein by reference. 

[0069] Referring to Fig. 1 9, there is shown a flowchart for a presently preferred embodiment of a shape enhancing 
40 filter 650 according to the present invention. At a source control point determination step 1910, the source control 
points 1810 (see Fig. 18) that are used for warping the image are determined by the feature points 420 (see Fig. 4) 
delineating the facial feature (e.g., eye) to be shape enhanced. In a destination control point determination step1920, 
the destination control points 1820 are determined. The location of destination control points 1820 defines the new 
location of the source control points 1810 in the shape modified image. The destination control points 1820 (D1 , D2, 
45 D3, and D4) are respectively the new locations of the source control points 1810 (S1 , S2, S3, and S4). In a warping 
step 1 930, the source and destination control points are used to warp the image. The location of the destination control 
points 1 820 are defined by the desired change of the facial feature shape. For example, if it is desired to increase the 
size of the eyes the destination control points 1820 are positioned as shown in Fig. 18. 

[0070] In a presently preferred embodiment, the parameters of the shape enhancing filter are used to define whether 
so the facial feature shape is increased or decreased by specifying the location of the destination control points 1820. 
The shape enhancing filtercan be incorporated into the GUI screen 78 (see Fig. 1 B) through an additional slider control. 
Moreover, there may be a unique shape enhancing slider for each facial feature. The system maps the position of the 
slider into appropriate parameter values of the shape enhancement filters. Moving the slider in one direction causes 
the facial feature to decrease in size while moving the slider in the opposite direction causes the facial feature to 
55 increase in size. The default position of the respective slider thus could be a neutral position not effecting any shape 
enhancement (until the slider is moved one way or the other). 

[0071] Throughout the foregoing description, certain parameters have been specified as candidates for system de- 
fault parameters, which, e.g., determine the initial settings of the enhancement filters and the initial settings of the 
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SSSSTT ^ th6 9raphiCal userin,erface Th «e parameters have been selected without limitation as 

2 ° ■PW»«» default parameters and should not be seen as a definitive or limiting set of para™ ters .t 

£££ZZ?£Z h 7 " tHiS ^ th8t ^ ° ,her P arame ^ ^'"ding others cLd in thte S P "on 
could be chosen and/or designated as default parameters. 



Claims 



10 



15 



20 



25 



1. A method for enhancing an appearance of a face located in a digital image, said method comprising the steps of: 
(a) acquiring a digital image containing one or more faces- 

l^fT l0Cat ; 0n ° f facial feature P° ints in the ™e or more faces, said facial feature points including 
points identifying salient features including one or more of skin, eyes, eyebrows, nose, mouth and hair 
(^usjngthe location of the facial feature points to segment the face into different regions said different regions 
ndudmg one or more of skin, eyes, eyebrows, nose, mouth, neck and hair regions- 9 

(d) determining one or more facially relevant characteristics of the different regions-' 

(e) based on the facially relevant characteristics of the different regions, selecting ( 1 ) two or more enhancement 
LrTaTd ^ *" * < 2) Mautt ^ ameters «* * e 

fmmt^dK !m e a n g h e anCemen, °" an enhanCed ^ 

2 - T n LT:^L^;" in claim 1 further comprisins the step ° f (9) app,yi ^ an ^ the 

3 " ^TS^IE? " C ' aim 1 Wher6in enhanCement ,ifters inc,ude * —t two of texture, skin tone, eye, 

4. The method as claimed in claim 1 wherein a gender classification algorithm is applied to determine the aender of 
the faces prior to the selection of the enhancement filters in step (e). eierm.ne the gender of 

5. The method as claimed in claim 1 wherein an.age classification algorithm is applied to determine the aqe of the 
faces pnor to the selection of the enhancement filters in step (e). ^ermine me age ot the 

6. The method as claimed in claim 1 wherein a distance between the eyes of the face is used to determine the default 
parameters for one or more of the enhancement filters selected in step (e). ^ermine me default 

? " In a7o e crilvice C,aimed **" * ^ UtNiZati ° n ^ h Step < 9 > inc,udes P"' ntin 9 th * ^ 

B In h t7ocaTd d evfce aimed C ' aim 2 Wherei " ^ Pr0C6SS aPP ' ied Step (9) inCludes archivin 9 the ^ge 

« 9 ' M aTe^ote Te^t^ * ^ * *" P™** aPP " ed St6p (9) inc,udes P rintjn 9 the ^ 

10 " in claim 2 wherein the uti,i2ation process applied in s,ep (9) inc,udes archK ' in9 the 

» 1 1 . The method as claimed in claim 1 wherein the facial characteristics determined in step (d) include at least one 
region, a gender of a subject in the image, and an age of the subject. particular 

55 1 2 ' claTr 1 ^ St ° ra9e m6diUm haVin9 inStmCti0nS Stored therein for causi "9 a computer to perform the method of 
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